Running Your Applications with Ease
2023-04-05
Welcome!
In this workshop, we will go over using containers on HPC resources, like UCLA’s Hoffman2
This is a Part I of my workshop on Containers.
WS_container.pdfWS_container.qmdContainers allow you to:
Containers can be:
Image Ref: Hoffman2 source, Stampede2 source
Containers provide a lightweight, portable, and consistent environment across different platforms.
To Understand Containers, we will first talk about Virtualization
GIF from https://giphy.com/
Software runs directly on OS from the physical hardware
Typical applications are in this fashion
module load softwareApplications running inside of a VM are running on a completely different set of (virtual) resources
A “Machine” within a “Machine”
Applications running inside of a container are running with the SAME kernal and physical resources as the host OS
A “OS” within a “OS”
HPC resources (like Hoffman2) are SHARED resources 👥
Great for easily installing software with apt/yum 📦
Great if your software requires MANY dependencies that would be complex installing on Hoffman2. ⛓️
Podman 📦
Docker 🐳
Security considerations 🛡️
To use Apptainer on Hoffman2, simply load the module:
Common Apptainer commands:
apptainer exec [options] container.sif
apptainer exec mypython.sif python3 test.py
# Runs the command `python3 test.py` inside the containerNote
Apptainer will NOT run on Hoffman2 login nodes.
Create 🛠️
Transfer ↪️
Run ▶️
1. Create 🛠️
Transfer
Run
$H2_CONTAINER_LOCCreate
Transfer
Run ▶️
Run Apptainer on your container:
Or run as a Batch (qsub) job
Create job script myjob.job
So….
Turns into
You can find the workshop material here:
-This example will use Tensorflow
We will use TensorFlow to train a model from this dataset
EX1 directorytf-example.py
IT DOESN’T WORK!!! Need tensorflow installed!!!
tensorflow_2.7.1.sif
Tip
apptainer shell [container.sif]apptainer exec [container.sif] [command]EX2 directory
pytorch_gpu.py fileLet’s run python3 tf-example.py on a GPU node
--nv option.
In this example, we’ll run a parallel MPI container using NWChem, a popular computational chemistry application.
Many applications use MPI to run across multiple CPUs, and NWChem is one of them.
$H2_CONTAINER_LOC/h2-nwchem_7.0.2.sifTypically, we will run NWChem like this:
To run inside the container:
mpirun in front of apptainer execmodule load intel/2022.1.1
`which mpirun` apptainer exec $H2_CONTAINER_LOC/h2-nwchem_7.0.2.sif nwchem water.nw > water.outA example batch job is located in EX3/nwchem.job
Questions? Comments? 🤔
Charles Peterson cpeterson@oarc.ucla.edu